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ABSTRACT 



In spring 1996, Florida's Pensacola Junior College (PJC) 
undertook the fourth assessment of institutional effectiveness using an 
assessment plan developed between 1990 and 1993. Outcomes were examined for 
the 1995-96 academic year with respect to the 51 goals, 147 objectives, and 
253 indicators of effectiveness identified by the assessment plan in 16 
functional areas. Area supervisors were requested to report on at least one 
objective and one indicator for each of the goals. Results of the process 
included the following: (1) 97 of the 253 indicators were selected to be 
reported for 1995-96, although with multiple reports for some indicators a 
total of 233 indicator reports were received; (2) 90.1% (n=210) of the 

indicators were shown to have been fully met in terms of designated levels of 
performance; and (3) indicators that did not meet standards included 
increasing annual student contacts by 2%, correctly placing 90% of first- time 
in college students, achieving a 5% annual increase in students, and 
including grade distributions in faculty evaluations. It was concluded that 
the central mission of the college was affirmed by the results of the 
assessment and it was recommended that the assessment be continued in 
1996-97; that the number of indicators be increased with special emphasis on 
those indicators that relate to outcome productivity measures rather than 
measures of internal processes. Tables showing outcomes by indicator and 
indicators that were not reported are included. (HAA) 
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Introduction: In an on-going effort to improve college operations and to improve the 
documentation of institutional effectiveness, a plan of assessment was developed in 1990-1993. In 
the spring of 1993, an initial run of the Pensacola Junior College Institutional Effectiveness Plan was 
undertaken for at least a single indicator and objective for each goal in the plan. In the spring of 
1994, the second institutional effectiveness assessment was made according to this plan. A third 
assessment was made in the spring of 1995. Following are the results of the fourth institutional 
effectiveness assessment undertaken in the spring of 1996. It must be noted that this report is not 



a final product, only the fourth step of an evolving process. 

Index: Executive Summary page 2 

Analysis page 3 

Conclusions page 4 

Recommendations page 5 

Comprehensive Goal, Objective, Indicator 

and Use Statements, TABLE I page 6 

Analysis of Institutional Effectiveness Indicators, TABLE II page 16 

List of Unreported Indicators Through 1995-1996, TABLE III page 17 



A Word of Caution: These indicators are intended to be data for affirmation of the mission of the 
college and for continued improvement of the institution. The success of the process requires a non- 
threatening environment in which the aggressive pursuit of evaluative data, both positive and 
negative, is encouraged. Consequently the achievement or non-achievement of a given indicator 
standard, while very important to the operational unit, is not as important to the purpose of 
institutional effectiveness assessment as is the fact that the college has examined that facet of the 
college’s productivity, has generated concrete information concerning it, and has proceeded to act 
on that information for the long-term improvement of the college. 
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EXECUTIVE SUMMARY 

Year 4 Institutional Effectiveness Assessment (1995 - 1996) 
(R-96-516, 7-31-96) 



The fourth institutional effectiveness assessment was undertaken by the Office of Institutional 
Research and Effectiveness in the spring of 1996, for the 1995-1996 academic year. This used the 
plan developed by the Institutional Effectiveness Taskforce between 1990 and 1993, which 
consisted in the fourth year evaluation of sixteen functional areas, 51 goals, 147 objectives and 253 
possible indicators. For the fourth run, area supervisors were requested to select one objective and 
one indicator for each of the goals, although some selected more and others less. Reports of 
performance indicators for these objectives were received through July 31, 1996. Several follow-up 
efforts were made. The process was successfully completed this year for the fourth time. 

Findings: 

(1) A total of 233 indicator reports were received for 1995-1996 (some indicator standards were 
used by more than one unit.) 

(2) Of these, 210 or 90.1% were fully met in terms of designated levels of performance. 

(3) Twenty-three did not meet the specified standards. 

(4) Data from these assessments were used to a high degree (96.6%). 

Conclusions: 

The central mission of the college is affirmed by the results of this assessment, unit-by-unit 
across the college. 

Recommendations: 

(1) The process must be continued in 1996-1997, at which time a general evaluation of the entire 
five-year process cycle will be conducted. 

(2) Continued simplification of the process is in order; especially in the reduction of the number 
of indicators. 

(3) Each institutional unit should strive to focus more of its institutional effectiveness efforts 
upon indicators that show basic college productivity respective of its mission, rather than 
process activity measures. Although processes of the college are important they represent 
means toward ends rather than true outcome productivity. 



ANALYSIS 



1. It is given under the current plan that each functional area (n=16) can have multiple goals 
(n=51), which in turn may have multiple objectives (n=147) and multiple indicators (n=253). 
In addition there is the potential for multiple reports based on individual indicators. In 1995- 
1996 assessment, there was to have been an indicator reported for each goal; that would have 
been a minimum 51 indicator responses. There were, in fact, 97 individual indicators actually 
selected to be reported. Some received multiple responses for the same objective, some were 
single responses for a total of 233 indicator reports in all. Thus, the indicator reporting rate was 
38.3% for 1995-1996 (that is, the percentage of individual standards reported at least one time). 
A summary of these reports is presented in TABLE I. 

2. TABLE II gives detailed information on these data. Of the 233 total indicator standard reports 
received, 210 or 90.1% of the prescribed standards were met or exceeded. Of all the reports 
received, the reporters stated in 225 or 96.6% of these instances that the report data (whether the 
standard was achieved or not) was used by the reporting unit for its improvement. Additional 
non-standard versions of standing indicators and new alternate indicators were reported by some 
units. 

3. TABLE III displays indicators that have yet to be examined (that is, not reported in 1992-1993, 
1993-1994, 1994-1995, or 1995-1996) but which must be assessed at least once during the five- 
year cycle of evaluation. 
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CONCLUSIONS 



It is important to recognize that this evaluative process is intended to assess the assessment process 

itself as well as to assess college performance each year. 

1. Qyerall indicator response suggests that PJC units are meeting or exceeding prescribed 
performance levels for the objectives assessed: and as a result, are meeting college mission 
goals . 

2. This fourth run was adequate. The level of cooperation by the participating reporters was 
excellent in the fourth run. The reporting system and paper trail worked well and continued to 
have a minimal impact on college resources. 

3. Too many indicators were found to be process rather than groducj - oriented. Virtually no (3) 
product indicators were chosen this year. More product outcome measures are needed. 
However, the product emphasis on the new institutional accountability process, the 
Performance Incentive Funding effort and the new Performance-Based Budgeting system more 
than make up for any product shortcoming in this effectiveness process. 

4. A continuing overreliance on satisfaction surveys was noted. There should be a curtailment of 
survey-based indicators and more emphasis placed on outcome measures of a quantitative 
nature when the second five-year cycle is planned. 

5. Subtle changes in some objectives and indicators continued to be made in the course of the 
process. This resulted in some indicator objectives having evolved into objectives more 
appropriate to unit operational plans or institution strategic plan objectives. Most of these were 
subjected to the formal review and approval process. Other unit leaders made major 
modifications of standard indicator measures. Still others elected to use new alternative 
indicator measures. 

6. Little opposition to the process was noted, though lack of full understanding of the process and 
its purpose is frequently encountered. (Some difference in opinion seems to exist between those 
who see the process as a mirror of SACS Cnteria and those who envision it as a test of college 
mission compliance.) Some see the process as an added burden. Others confuse it with Strategic 
Planning , the SACS reaffirmation processes and the state accountability process. 
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RECOMMENDATIONS 



That: 

1 . The process be continued, simplified and refined. 

2. College leadership continue to emphasize the importance of the process. 

3. In 1996-1997, the total number of indicator reports must be increased, with special emphasis 
on those indicators that relate to outcome productivity measures rather than measures of 
internal processes (see TABLE III). This is the last opportunity in this cycle for units to 
complete their planned indicators. 

4. Individual reporting units continue to utilize the results of these assessments for continuous 
improvement. 

5. Continued efforts must be made to educate PJC personnel on the institutional effectiveness 
process. 

6. In 1996-1997, begin planning preliminary work for the new 1997-2002 institutional 
effectiveness cycle. 
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TABLE O. 



ANALYSIS OF REPORTED INSTITUTIONAL EFFECTIVENESS INDICATORS (a) 

1995-1996 
(YEAR IV) 



TOTAL INDICATOR REPORTS RECEIVED 


233 


100.0% 


PRODUCTIVITY INDICATORS REPORT RECEIVED 


3 


1.3% 


PRODUCTIVITY INDICATORS REPORTS ACHIEVED 


100.0% 


3 


1.3% 


PROCESS INDICATOR REPORTS RECEIVED 


230 


98.7% 


PROCESS INDICATORS ACHIEVED 


90.0% 


207 


88.8% 


ALL INDICATORS ACHIEVED 


207 


88.8% 


INDIVIDUAL INDICATORS DATA USED (of a possible 252) 


97 


38.3% 



(a) Source: Tabulated from TABLE I. data. 
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TABLE m. 

LIST OF UNREPORTED INDICATORS THROUGH 1995-1996* 



New I.A.l.b. 


IV.C.4.b. 


xra.c.i.c. 


XVI. E. 3. b. 


New I.A.l.c. 


IV.D.2.a. 


xm.c. 2 .b. 


XVI.E.3.C. 


New I.A.l.d. 


IV.D.3.a. 


xni.c.2.c. 


XVI.F.l.b. 


New I.A.3.a. 


IV.D.3.b. 


IX.A.3.a. 


XVI.F.l.c 


New I.A.3.b. 


V.A.l.a. 


IX.A.5.a. 


XVI. F. 2. a. 


New I.E.2.a. 


V.B.l.a.(l) 


XI.B.l.a. 


XVI.H.3.a. 


New XVI. H. 2. b. 


V.B.l.a.(3) 


Xm.C.3.a. 




New XVI.B.2.b. 


V.C.l.c. 


xm.c.3.b. 




I.C.l.b. 


V.C.l.e. 


XIII.C.4.a. 




I.C.2.a. 


V.C.2.a. 


xm.c.4.b. 




I.D.l.b. 


V.C.4.a. 


XTV.C.2.a. 




II. Ala. 


VI.A.5.a. 


XIX.C.2.a. 




in.A.l.b. 


IX.A.3.a. 


XIX.C.3.b. 




m.A.l.c. 


IX.A.5.a. 


XIV.C.4.a. 




III. A. Id. 


XI.B.l.a. 


XV.A.3.a. 




III. Ale. 


XII.A.2.C. 


XV.A.5.b. 




III.A.2.a. 


XII.B.2.a. 


XVI.C.l.b. 




III.A.2.b. 


xm.A.i.b. 


XVI. C. 2. a. 




III.A.2.b. 


xm.A.i.c. 


XVI. C. 4. a. 




III. A. 2. d. 


Xin.A.3.a. 


XVI. C. 5. a. 




IV.A.2.a. 


XIII.A.3.b. 


XVI. C. 5. b. 




IV.A.3.d. 


Xm.A.4.a. 


XVI.D.l.a. 




IV.A.3.e. 


Xm.A.4.b. 


XVI.D.l.b. 




IV.B.l.a. 


Xin.A.4.c. 


XVI. D. 3. a. 




IV.B.3.b. 


XIII.A.5.a. 


XVI.E.l.b. 




IV.B.3.d. 


XIII.A.5.b. 


XVI.E.2.a. 




IV.B.3.e. 


Xin.A.6.b. 


XVI.E.2.b. 




IV.C.3.a. 


XIII. B. 2. a. 


XVI.E.3.a. 
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